suppressMessages({
library(tidyverse)
library(brms)
library(posterior)
library(here)
library(tinytable)
})Ordinal treatment effect of mechanical thrombectomy
Comprehensive evaluation of randomized evidence for mechanical thrombectomy using bayesian implementation of random effects meta-analysis with ordinal data in stan using brms.
Dependencies
Data
all_dat <- readRDS("all_dat.rds")Data description
Table 1 shows the data from the 27 randomized trials of thrombectomy. Trials are categorized by type of stroke treated.
| mRS at six months | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| type | trial | treatment | zero | one | two | three | four | five | six |
| large | angel | thrombectomy | 9 | 19 | 41 | 39 | 45 | 27 | 50 |
| large | angel | medical | 0 | 8 | 18 | 49 | 60 | 45 | 45 |
| large | rescue | thrombectomy | 2 | 3 | 9 | 17 | 33 | 18 | 18 |
| large | rescue | medical | 0 | 3 | 5 | 5 | 25 | 40 | 24 |
| large | select | thrombectomy | 2 | 9 | 25 | 31 | 27 | 15 | 68 |
| large | select | medical | 0 | 3 | 9 | 20 | 36 | 32 | 71 |
| large | tension | thrombectomy | 5 | 5 | 11 | 18 | 24 | 15 | 46 |
| large | tension | medical | 1 | 1 | 1 | 13 | 16 | 24 | 66 |
| large | tesla | thrombectomy | 6 | 8 | 8 | 23 | 30 | 23 | 53 |
| large | tesla | medical | 2 | 6 | 5 | 16 | 35 | 33 | 49 |
| early | escape | thrombectomy | 25 | 34 | 30 | 26 | 21 | 11 | 16 |
| early | escape | medical | 10 | 15 | 18 | 22 | 35 | 18 | 28 |
| early | extend | thrombectomy | 9 | 9 | 7 | 6 | 1 | 0 | 3 |
| early | extend | medical | 6 | 4 | 4 | 4 | 6 | 4 | 7 |
| early | mrclean | thrombectomy | 7 | 21 | 49 | 42 | 51 | 14 | 49 |
| early | mrclean | medical | 1 | 16 | 35 | 43 | 80 | 32 | 59 |
| early | piste | thrombectomy | 5 | 9 | 3 | 4 | 4 | 1 | 7 |
| early | piste | medical | 1 | 5 | 6 | 7 | 4 | 3 | 4 |
| early | resilient | thrombectomy | 9 | 13 | 17 | 24 | 14 | 7 | 27 |
| early | resilient | medical | 3 | 7 | 13 | 17 | 21 | 18 | 33 |
| early | revascat | thrombectomy | 7 | 18 | 20 | 19 | 8 | 12 | 19 |
| early | revascat | medical | 6 | 7 | 16 | 20 | 17 | 21 | 16 |
| early | swift | thrombectomy | 17 | 25 | 17 | 12 | 15 | 3 | 9 |
| early | swift | medical | 8 | 10 | 15 | 16 | 20 | 12 | 12 |
| early | therapy | thrombectomy | 3 | 10 | 6 | 8 | 15 | 2 | 6 |
| early | therapy | medical | 1 | 6 | 7 | 7 | 10 | 4 | 11 |
| early | thrace | thrombectomy | 31 | 39 | 36 | 25 | 34 | 11 | 24 |
| early | thrace | medical | 24 | 33 | 28 | 25 | 56 | 9 | 27 |
| late | dawn | thrombectomy | 10 | 24 | 18 | 14 | 14 | 8 | 20 |
| late | dawn | medical | 4 | 5 | 4 | 16 | 34 | 19 | 18 |
| late | defuse3 | thrombectomy | 9 | 15 | 17 | 14 | 17 | 7 | 13 |
| late | defuse3 | medical | 7 | 4 | 4 | 14 | 24 | 14 | 23 |
| late | mrcleanlate | thrombectomy | 23 | 31 | 46 | 28 | 31 | 33 | 61 |
| late | mrcleanlate | medical | 5 | 35 | 44 | 20 | 27 | 42 | 74 |
| late | positive | thrombectomy | 4 | 4 | 1 | 0 | 1 | 1 | 1 |
| late | positive | medical | 1 | 3 | 5 | 3 | 2 | 3 | 4 |
| basilar | attention | thrombectomy | 11 | 34 | 29 | 29 | 11 | 27 | 84 |
| basilar | attention | medical | 5 | 5 | 3 | 14 | 6 | 19 | 63 |
| basilar | baoche | thrombectomy | 7 | 20 | 17 | 8 | 10 | 15 | 34 |
| basilar | baoche | medical | 1 | 6 | 7 | 11 | 20 | 16 | 45 |
| basilar | basics | thrombectomy | 8 | 19 | 27 | 14 | 10 | 17 | 59 |
| basilar | basics | medical | 6 | 13 | 25 | 11 | 16 | 12 | 63 |
| basilar | best | thrombectomy | 4 | 11 | 7 | 6 | 5 | 11 | 22 |
| basilar | best | medical | 6 | 10 | 2 | 3 | 12 | 7 | 25 |
| medium | escapemevo | thrombectomy | 45 | 61 | 32 | 46 | 28 | 9 | 34 |
| medium | escapemevo | medical | 38 | 80 | 43 | 52 | 26 | 12 | 23 |
| medium | distal | thrombectomy | 36 | 58 | 59 | 47 | 26 | 3 | 42 |
| medium | distal | medical | 47 | 54 | 46 | 57 | 24 | 3 | 38 |
| historical | ims3 | thrombectomy | 53 | 69 | 55 | 71 | 64 | 20 | 83 |
| historical | ims3 | medical | 19 | 39 | 28 | 35 | 30 | 15 | 48 |
| historical | synthesis | thrombectomy | 22 | 33 | 21 | 37 | 32 | 10 | 26 |
| historical | synthesis | medical | 28 | 35 | 21 | 28 | 38 | 13 | 18 |
| historical | mrrescue | thrombectomy | 1 | 3 | 3 | 13 | 18 | 13 | 12 |
| historical | mrrescue | medical | 1 | 5 | 4 | 13 | 15 | 9 | 7 |
“Large” refers to anterior circulation strokes with a large infarct core. “Early” refers to anterior circulation strokes with a small core, treated in an early time window. “Late” refers to anterior circulation strokes with a small core, treated in an extended time window. “Basilar” refers to posterior circulation strokes due to vertebro-basilar occlusion. “Medium” refers to anterior circulation strokes due to medium vessel occlusion.
Data exploration
Figure 1 plots the data from Table 1.
Model
Model description
We use brms to define and fit an ordinal meta-regression in the probabilistic programming language stan. Specifically, we use an adjacent category ordinal regression with category specific treatment effects, correlated trial-specific varying intercepts and slopes, and a predictor term for stroke type.
Model summary
Summary of the model is shown as follows.
Family: acat
Links: mu = logit; disc = identity
Formula: ordinal_value ~ type + cs(treatment) + (1 + treatment | trial)
Data: all_dat_long (Number of observations: 7695)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Group-Level Effects:
~trial (Number of levels: 27)
Estimate Est.Error l-95% CI u-95% CI Rhat
sd(Intercept) 0.15 0.03 0.09 0.22 1.00
sd(treatmentthrombectomy) 0.13 0.03 0.09 0.19 1.00
cor(Intercept,treatmentthrombectomy) -0.57 0.25 -0.90 0.06 1.01
Bulk_ESS Tail_ESS
sd(Intercept) 1195 1878
sd(treatmentthrombectomy) 1435 2346
cor(Intercept,treatmentthrombectomy) 738 1339
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
Intercept[1] -1.03 0.10 -1.22 -0.83 1.00 2151
Intercept[2] -0.39 0.09 -0.57 -0.21 1.00 1941
Intercept[3] -0.60 0.09 -0.77 -0.43 1.00 2049
Intercept[4] -0.53 0.08 -0.70 -0.37 1.00 1693
Intercept[5] 0.14 0.08 -0.03 0.31 1.00 1959
Intercept[6] -0.81 0.08 -0.97 -0.65 1.00 1609
typeearly -0.36 0.07 -0.50 -0.23 1.00 1368
typelate -0.30 0.08 -0.46 -0.15 1.00 1844
typebasilar -0.09 0.08 -0.25 0.06 1.00 1625
typemedium -0.53 0.14 -0.79 -0.26 1.00 943
typehistorical -0.31 0.11 -0.54 -0.09 1.01 936
treatmentthrombectomy[1] -0.21 0.11 -0.42 -0.01 1.00 3488
treatmentthrombectomy[2] -0.09 0.09 -0.27 0.10 1.00 3039
treatmentthrombectomy[3] -0.31 0.09 -0.48 -0.13 1.00 3461
treatmentthrombectomy[4] -0.34 0.09 -0.52 -0.17 1.00 3141
treatmentthrombectomy[5] -0.21 0.10 -0.40 -0.02 1.00 3472
treatmentthrombectomy[6] 0.36 0.09 0.18 0.54 1.00 3282
Tail_ESS
Intercept[1] 2747
Intercept[2] 2918
Intercept[3] 2557
Intercept[4] 2279
Intercept[5] 2236
Intercept[6] 2191
typeearly 1952
typelate 2256
typebasilar 2545
typemedium 1951
typehistorical 1389
treatmentthrombectomy[1] 3015
treatmentthrombectomy[2] 2420
treatmentthrombectomy[3] 2633
treatmentthrombectomy[4] 2792
treatmentthrombectomy[5] 3087
treatmentthrombectomy[6] 2985
Family Specific Parameters:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
disc 1.00 0.00 1.00 1.00 NA NA NA
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
Model diagnostics
Because the model is fit to the data using MCMC, it is important assess for any errors during the sampling process.
Figure 2 shows the trace plots for each parameter of the model. Trace plots show good exploration of the parameter space.
There were no divergences during model fit. Figure 3 shows rhat values all less than 1.05.
Figure 4 shows that the estimated ratio of effective sample sizes to total sample sizes was greater than 0.1 for all parameters.
Model checks
In addition to verifying that there were no technical problems with the MCMC algorithm when fitting the model, we also need to check for evidence of over-fitting or under-fitting to the data.
Figure 5 is a graphical posterior predictive check of the model fit to the observed data. This can be thought of as the bayesian analogue of a chi-squared goodness-of-fit test. The posterior predictive distributions of the model (yrep) all include the observed values (y). There is no obvious evidence of misfit of model predictions to data.
We can also check the predictive performance of the model using leave-one-out cross-validation.
Computed from 4000 by 7695 log-likelihood matrix
Estimate SE
elpd_loo -14045.5 39.8
p_loo 54.9 0.7
looic 28090.9 79.6
------
Monte Carlo SE of elpd_loo is 0.1.
All Pareto k estimates are good (k < 0.5).
See help('pareto-k-diagnostic') for details.
There are no signs of misspecification: estimated parameters closely match actual parameters, Monte Carlo standard error of the expected log posterior density is low, and all Pareto K estimates are less than 0.5.
Results
Ordinal treatment effect of thrombectomy across stroke types
The model considers a hypothetical patient enrolled in a new randomized trial of thrombectomy and uses the totality of observed data from previous trials to generate expected probabilities for each mRS at six months if the patient were enrolled in the medical arm of the trial, as well as the change in probability (either increase or decrease) if the patient was instead enrolled in the treatment arm of the trial. We refer to these changes as the ordinal treatment effect.
Large stroke
Table 2 shows the expected ordinal probabilities for a hypothetical patient randomized to the medical treatment arm of a new large core stroke trial.
| estimate | |||
|---|---|---|---|
| mRS | mean | low | high |
| 0 | 1.6 | 0.9 | 2.7 |
| 1 | 4.4 | 2.8 | 6.5 |
| 2 | 6.5 | 4.6 | 8.6 |
| 3 | 11.7 | 9.5 | 14.0 |
| 4 | 19.8 | 18.0 | 21.6 |
| 5 | 17.2 | 15.5 | 19.0 |
| 6 | 38.8 | 31.5 | 45.6 |
Table 3 shows the expected ordinal treatment effect of thrombectomy in this trial.
| estimate | |||
|---|---|---|---|
| mRS | mean | low | high |
| 0 | 1.8 | 0.9 | 3.0 |
| 1 | 3.2 | 1.7 | 4.9 |
| 2 | 3.8 | 2.2 | 5.4 |
| 3 | 1.9 | 0.2 | 3.6 |
| 4 | -3.4 | -5.5 | -1.4 |
| 5 | -5.7 | -7.6 | -3.8 |
| 6 | -1.6 | -5.8 | 2.8 |
Small stroke, early window
Table 4 shows the expected ordinal probabilities for a hypothetical patient randomized to the medical treatment arm of a new small stroke early window trial.
| estimate | |||
|---|---|---|---|
| mRS | mean | low | high |
| 0 | 6.6 | 4.7 | 9.0 |
| 1 | 12.8 | 10.1 | 15.8 |
| 2 | 13.2 | 11.2 | 15.1 |
| 3 | 16.7 | 15.2 | 18.1 |
| 4 | 19.8 | 18.2 | 21.4 |
| 5 | 12.0 | 10.1 | 14.0 |
| 6 | 18.9 | 14.7 | 23.4 |
Table 3 shows the expected ordinal treatment effect of thrombectomy in this trial.
| estimate | |||
|---|---|---|---|
| mRS | mean | low | high |
| 0 | 5.2 | 3.2 | 7.5 |
| 1 | 5.7 | 3.3 | 8.2 |
| 2 | 4.4 | 2.4 | 6.3 |
| 3 | -0.4 | -2.3 | 1.5 |
| 4 | -6.0 | -8.0 | -4.1 |
| 5 | -5.3 | -6.9 | -3.7 |
| 6 | -3.7 | -6.7 | -0.8 |
Small stroke, late window
Table 6 shows the expected ordinal probabilities for a hypothetical patient randomized to the medical treatment arm of a new small stroke late window trial.
| estimate | |||
|---|---|---|---|
| mRS | mean | low | high |
| 0 | 5.4 | 3.2 | 8.4 |
| 1 | 11.0 | 7.5 | 15.1 |
| 2 | 12.0 | 9.3 | 14.7 |
| 3 | 16.1 | 14.2 | 17.8 |
| 4 | 20.3 | 18.5 | 21.9 |
| 5 | 13.1 | 10.4 | 15.7 |
| 6 | 22.0 | 15.4 | 29.0 |
Table 7 shows the expected ordinal treatment effect of thrombectomy in this trial.
| estimate | |||
|---|---|---|---|
| mRS | mean | low | high |
| 0 | 4.6 | 2.5 | 7.0 |
| 1 | 5.5 | 3.2 | 7.9 |
| 2 | 4.5 | 2.6 | 6.4 |
| 3 | 0.2 | -1.9 | 2.3 |
| 4 | -5.7 | -7.7 | -3.7 |
| 5 | -5.5 | -7.2 | -3.8 |
| 6 | -3.6 | -6.9 | -0.4 |
Basilar stroke
Table 8 shows the expected ordinal probabilities for a hypothetical patient randomized to the medical treatment arm of a new small stroke late window trial.
| estimate | |||
|---|---|---|---|
| mRS | mean | low | high |
| 0 | 2.4 | 1.3 | 4.0 |
| 1 | 6.1 | 3.7 | 8.9 |
| 2 | 8.1 | 5.7 | 10.6 |
| 3 | 13.3 | 10.9 | 15.5 |
| 4 | 20.5 | 18.9 | 22.1 |
| 5 | 16.3 | 14.1 | 18.2 |
| 6 | 33.3 | 26.2 | 41.0 |
Table 9 shows the expected ordinal treatment effect of thrombectomy in this trial.
| estimate | |||
|---|---|---|---|
| mRS | mean | low | high |
| 0 | 2.5 | 1.3 | 4.2 |
| 1 | 4.0 | 2.2 | 6.0 |
| 2 | 4.2 | 2.6 | 6.0 |
| 3 | 1.6 | -0.2 | 3.3 |
| 4 | -4.2 | -6.3 | -2.1 |
| 5 | -5.7 | -7.6 | -4.0 |
| 6 | -2.5 | -6.5 | 1.8 |
Medium stroke
Table 10 shows the expected ordinal probabilities for a hypothetical patient randomized to the medical treatment arm of a new medium stroke trial.
| estimate | |||
|---|---|---|---|
| mRS | mean | low | high |
| 0 | 11.4 | 5.4 | 18.9 |
| 1 | 18.2 | 11.1 | 25.2 |
| 2 | 15.7 | 11.9 | 18.3 |
| 3 | 16.8 | 14.9 | 18.3 |
| 4 | 17.0 | 12.5 | 20.7 |
| 5 | 8.9 | 5.2 | 13.2 |
| 6 | 12.1 | 5.6 | 21.6 |
Table 11 shows the expected ordinal treatment effect of thrombectomy in this trial.
| estimate | |||
|---|---|---|---|
| mRS | mean | low | high |
| 0 | 6.8 | 3.6 | 10.5 |
| 1 | 5.5 | 2.5 | 8.4 |
| 2 | 3.2 | 0.4 | 5.7 |
| 3 | -1.9 | -4.2 | 0.7 |
| 4 | -6.2 | -8.1 | -4.4 |
| 5 | -4.3 | -6.2 | -2.6 |
| 6 | -3.1 | -5.7 | -1.0 |
Summary
Conditional on the observed data and the model used to analyze the data, thrombectomy is expected to reliably lower the probability of bad outcomes and raise the probability of good outcomes across all stroke types. Figure 6 plots these estimates to show the trend graphically.
These treatment effects should be considered in the context of the expected probability of each outcome with medical treatment alone, as shown in Figure 7, which shows considerable variation by stroke type.
Absolute dichotomous effect
Table 12 shows the implied dichotomous treatment effect estimate from this model.
| estimate | |||
|---|---|---|---|
| type | mean | low | high |
| large | 8.8 | 5.2 | 12.7 |
| early | 15.3 | 10.7 | 20.1 |
| late | 14.5 | 9.9 | 19.5 |
| basilar | 10.8 | 6.7 | 15.2 |
| medium | 15.5 | 10.8 | 20.5 |
| historical | 14.6 | 9.8 | 19.7 |
Figure 8 plots the estimates with the observed effect for comparison.
Figure 9 plots the cumulative effects.
Sensitivity Analyses
Sensitivity to relative comparison versus absolute
Relative ordinal effect
Figure 10 shows the expected relative impact of thrombectomy on mRS in a new trial. Thrombectomy has a higher relative impact for large and basilar strokes because the base-rates of poor outcomes are high and the base-rates for good outcomes are low.
Relative dichotomous effect
Table 13 shows this effect in relative terms.
| estimate | |||
|---|---|---|---|
| type | mean | low | high |
| large | 1.7 | 1.4 | 2.1 |
| early | 1.5 | 1.3 | 1.7 |
| late | 1.5 | 1.3 | 1.8 |
| basilar | 1.7 | 1.4 | 2.0 |
| medium | 1.4 | 1.2 | 1.6 |
| historical | 1.5 | 1.3 | 1.8 |
Figure 11 plots the estimates with the observed effect for comparison.
Sensitivity to prior specification
With flat priors, there is no difference in out-of-sample performance.
elpd_diff se_diff
omr_ip 0.0 0.0
omr_flat -0.2 0.2
Figure 12 shows treatment effect estimate under flat prior assumption, which can be compared to Figure 6, the estimates under the main model specification.
The results are virtually identical. Conclusion: reported results are relatively insensitive to prior choice.
Sensitivity to adjacent category parameterization
The main model assumes the adjacent category paramerization of ordinal regression, which allows for the modelling of category specific effects. An alternative parameterization of ordinal regression, referred to as cumulative ordinal regression, makes the proportional odds assumption.
elpd_diff se_diff
omr_ip 0.0 0.0
omr_cum -37.8 9.1
In addition to making the questionable assumption that the treatment effect of thrombectomy is constant across all mRS categories, this model also results in worse out-of-sample performance.
Sensitivity to meta-regression
Dropping predictors for stroke type does not result in decreased out-of-sample predictive performance.
elpd_diff se_diff
omr_ip 0.0 0.0
oma -0.8 1.7
This implies that the ordinal treatment effect of thrombectomy is relatively consistent across stroke types, at least for this data set.